Cloud Computing for Science

Infrastructure-as-a-Service (IaaS) Cloud computing is emerging as a viable alternative to the acquisition and management of physical resources. The Nimbus project works towards enabling scientific communities to leverage cloud computing by providing (1) an IaaS toolkit allowing providers to configure clouds, (2) user-level tools for contextualization and scaling allowing users to easily take advantage of IaaS, and (3) an extensible open source implementation of both allowing users to customize the implementation to their needs. The talk will give an overview of the challenges and potential of cloud computing projects in science as well as the way in which the Nimbus project at Argonne National Laboratory addresses them. I will also describe what attracted various scientific communities to cloud computing and give examples of how they integrated this new model into their work. Finally, I will describe how scientific projects collaborating with us inspire technological development and take advantage of various Nimbus features

KheOps: Cost-effective Repeatability, Reproducibility, and Replicability of Edge-to-Cloud Experiments

Distributed infrastructures for computation and analytics are now evolving towards an interconnected ecosystem allowing complex scientific workflows to be executed across hybrid systems spanning from IoT Edge devices to Clouds, and sometimes to supercomputers (the Computing Continuum). Understanding the performance trade-offs of large-scale workflows deployed on such complex Edge-to-Cloud Continuum is challenging. To achieve this, one needs to systematically perform experiments, to enable their reproducibility and allow other researchers to replicate the study and the obtained conclusions on different infrastructures. This breaks down to the tedious process of reconciling the numerous experimental requirements and constraints with low-level infrastructure design choices.To address the limitations of the main state-of-the-art approaches for distributed, collaborative experimentation, such as Google Colab, Kaggle, and Code Ocean, we propose KheOps, a collaborative environment specifically designed to enable cost-effective reproducibility and replicability of Edge-to-Cloud experiments. KheOps is composed of three core elements: (1) an experiment repository; (2) a notebook environment; and (3) a multi-platform experiment methodology.We illustrate KheOps with a real-life Edge-to-Cloud application. The evaluations explore the point of view of the authors of an experiment described in an article (who aim to make their experiments reproducible) and the perspective of their readers (who aim to replicate the experiment). The results show how KheOps helps authors to systematically perform repeatable and reproducible experiments on the Grid5000 + FIT IoT LAB testbeds. Furthermore, KheOps helps readers to cost-effectively replicate authors experiments in different infrastructures such as Chameleon Cloud + CHI@Edge testbeds, and obtain the same conclusions with high accuracies (> 88% for all performance metrics)

On the Use of Cloud Computing for Scientific Workflows

This paper explores the use of cloud computing for scientific workflows, focusing on a widely used astronomy application-Montage. The approach is to evaluate from the point of view of a scientific workflow the tradeoffs between running in a local environment, if such is available, and running in a virtual environment via remote, wide-area network resource access. Our results show that for Montage, a workflow with short job runtimes, the virtual environment can provide good compute time performance but it can suffer from resource scheduling delays and wide-area communications

Hydroinformatics On The Cloud: Data Integration, Modeling And Information Communication For Flood Risk Management

The Iowa Flood Information System (IFIS) is a web-based platform developed by the Iowa Flood Center (IFC) to provide access to flood inundation maps, real-time flood conditions, flood warnings and forecasts, flood-related data, information and interactive visualizations for communities in Iowa. The key elements of the IFIS are: (1) flood inundation maps, (2) autonomous “bridge sensors” that monitor water level in streams and rivers in real time, and (3) real-time flood forecasting models capable of providing flood warning to over 1000 communities in Iowa. The IFIS represents a hybrid of file and compute servers, including a High Performance Computing cluster, codes in different languages, data streams and web services, databases, scripts and visualizations. The IFIS processes raw data (50GB/day) from NEXRAD radars, creates rainfall maps (3GB/day) every 5 minutes, and integrates real-time data from over 600 sensors in Iowa. Even though the IFIS serves over 75,000 users in Iowa using local infrastructure, cloud computing can improve scalability, speed, cost efficiency, accessibility, security, resiliency and uptime. In this collaborative study between the Iowa Flood Center and the Nimbus team at the Argonne National Laboratory, we have analyzed feasibility and price/performance measures of moving the MPI-based computations to the cloud as well as assessment of response times from our interactive web-based system. Moving the system to the cloud, and making it independent and portable, would enable us to share our model easily with the flood research community. This presentation provides an overview of the tools and interfaces in the IFIS, and transition of the IFIS from a local infrastructure to cloud computing environment

Evaluating Streaming Strategies for Event Processing across Infrastructure Clouds

International audienceInfrastructure clouds revolutionized the way in which we approach resource procurement by providing an easy way to lease compute and storage resources on short notice, for a short amount of time, and on a pay-as-you-go basis. This new opportunity, however, introduces new performance trade-offs. Making the right choices in leveraging different types of storage available in the cloud is particularly important for applications that depend on managing large amounts of data within and across clouds. An increasing number of such applications conformto a pattern in which data processing relies on streaming the data to a compute platform where a set of similar operations is repeatedly applied to independent chunks of data. This pattern is evident in virtual observatories such as the Ocean Observatory Initiative, in cases when new data is evaluated against existing features in geospatial computations or when experimental data is processed as a series of time events. In this paper, we propose two strategies for efficiently implementing such streaming in the cloud and evaluate them in the contextof an ATLAS application processing experimental data. Our results show that choosing the right cloud configuration can improve overall application performance by as much as three times

Evaluating Streaming Strategies for Event Processing across Infrastructure Clouds

Abstract-Infrastructure clouds revolutionized the way in which we approach resource procurement by providing an easy way to lease compute and storage resources on short notice, for a short amount of time, and on a pay-as-you-go basis. This new opportunity, however, introduces new performance trade-offs. Making the right choices in leveraging different types of storage available in the cloud is particularly important for applications that depend on managing large amounts of data within and across clouds. An increasing number of such applications conform to a pattern in which data processing relies on streaming the data to a compute platform where a set of similar operations is repeatedly applied to independent chunks of data. This pattern is evident in virtual observatories such as the Ocean Observatory Initiative, in cases when new data is evaluated against existing features in geospatial computations or when experimental data is processed as a series of time events. In this paper, we propose two strategies for efficiently implementing such streaming in the cloud and evaluate them in the context of an ATLAS application processing experimental data. Our results show that choosing the right cloud configuration can improve overall application performance by as much as three times

Scaling Smart Appliances for Spatial Data Synthesis

International audienceWith the rapidly growing number of dynamic data streams produced by sensing and experimental devices as well as social networks, scientists are given an unprecedented opportunity to explore a variety of environmental and social phenomena ranging from understanding of weather and climate to population dynamics. One of the main challenges is that dynamic data streams and their computation requirements are volatile: sensors or social networks may generate data at highly variable rates, processing time in an application may significantly change from one stage to the next one, or different phenomena may simply generate different levels of interest. Cloud computing is a promising platform allowing us to cope with such volatility because it enables us to allocate computational resources on demand, for short periods of time, and at an acceptable cost. At the same time using clouds for this purpose is challenging because an application may yield a very different performance depending on the hosting infrastructure, requiring us to pay special attention to how and where we schedule resources. In this poster, we describe our experiences using an application relying on input from social networks, notably geo-located tweets, to discover correlation between users’ work and home locations, with focus in the Illinois area. Our overall intent is to assess the impact of running the same application in offerings from different providers; to this end, we execute data filtering and per-user classification applications in two flavors of Chameleon cloud instances, namely bare-metal and KVM. Also, we analyze specific configuration parameters, such as data block size, replication factor and parallel processing, towards statistically modeling the application performance in a given infrastructure. We then identify and discuss the key parameters that influence the execution time. Finally, we look into the gains brought by accounting for data proximity when scheduling a resource in a multi-site environment

Supporting Experimental Computer Science

The ability to conduct consistent, controlled, and repeatable large-scale experiments in all areas of computer science related to parallel, large-scale, or distributed computing and networking is critical to the future and development of computer science. Yet conducting such experiments is still too often a challenge for researchers, students, and practitioners because of the unavailability of dedicated resources, inability to create controlled experimental conditions, and variability in software. Availability, repeatability, and open sharing of electronic products are all still difficult to achieve. To discuss those challenges and share experiences in their solution, the Workshop on Experimental Support for Computer Science brought together scientists involved in building and operating infrastructures dedicated to sup- porting computer science experiments to discuss challenges and solutions in this space. The workshop was held in November 2011 and was collocated with the SC11 conference in Seattle, Washington. Our objec- tives were to share experiences and knowledge related to supporting large-scale experiments conducted on experimental infrastructures, understand user requirements, and discuss methodologies and opportunities created by emerging technologies. This report ties together the workshop presentations and discussion and the consensus that emerged on the state of the field and directions for moving forward.La possibilité d'effectuer des expériences à grande échelle consistantes, contrôlées, et reproductibles dans tous les domaines de l'informatique liés au parallélisme et au calcul distribué est critique pour le futur et le développement de l'informatique. Le lancement de telles expérimentations est souvent difficile pour les chercheurs, les étudiants et les développeurs et ceci en partie à cause de l'absence de ressources dédiées, l'impossibilité de créer des conditions expérimentales contrôlées et l'évolution des logiciels. La disponibilité, la reproductibilité, et le partage ouvert de plates-formes informatiques sont toujours difficiles à atteindre. Afin de discuter de ces challenges et de partager nos expériences sur les solutions à ces problèmes, le workshop "Experimental Support for Computer Science" a réuni des scientifiques impliqués dans la construction et la maintenance de plates-formes expérimentales dédiées au support pour les expériences informatiques pour discuter des challenges et de leurs solutions. Ce workshop s'est tenu en novembre 2011 au sein de la conférence SC11 à Seattle, Washington. Notre objectif était de partager notre expériences et nos connais- sances autour du support pour les expériences à grande échelle lancées sur des plates-formes d'expérimentation, comprendre les besoins des utilisateurs et discuter des méthodes et des opportunités créées par ces technologies émergentes. Ce rapport présente les contributions liées aux présentations du workshop et aux discussions qui ont eu lieu et le consensus issu sur l'état de l'art et des directions pour les travaux futurs